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A UNKING CROSS BAR CONTROLLER 
Eltan MEDINA and David 8HEMLA 

FIELD OF THE INVENTION 

The present Invention relates to network switches generally and to 
5 cross-bars in particular. 

BACKGROUND OF THE INVENTION 

A network switch creates a network among a plurality of end nodes, 
such as wortetations, and other network switches connected thereto. Each end 
node iG connected to one port of the network. Ttie poils also sen/e to connect 

10 network switches together. 

Each end node sends packets of data to the network switch which the 
switch then routes either to another of the end nodes connected thereto or to a 
network switch to which the destination end node Is connected. In the latter case, 
the receiving network switch routes the padcet lo llie destination end node. 

15 Each network switch has to temporarily store the packets of data which 

it receives from the units (end node or network switch) connected to it while the 
switch determines how, when and through whteh port to retransrnil tlie packets. 
Each packet can be transmitted to only one destination address (a "unlcaef 
packet) or to more than one unit (a "multicast" or "broadcast" packet). For 

2D multicast and broadcast packets, the switch typically stores the packet only once 
and transmits multiple copies of the packet to some (multicast) or all (broadcast) 
of its ports, Once the packet has been transmitted lo all of its destinations, it can 
be removed from the memoiy or written over. 
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Switching Ethernet Controllers (SECs) are network switches that 
iniplement the Ethernet switching protocol. According to the protocol, the 
Qhernet network (cabling and hthemfit ports) operales at 10 Megabits per 
second. Switches which operate at the desired speed of 10 Megabits per second 

s are known as provldifig "fijli-wire" throughput. 

In US patent application 08/790,155, filed January 28. 1997, and 
incorporated heroin by reference, in order to optimise through-put time, 
communication between ShCs attempts to utilize Uie bus as little as possible so 
that the bus will be available as suon as a SEC wants to utilize it. Therefore, each 

10 SEC Includes a wiite-only bus communication unit which transfers the packets oul 
of the SEC by only writing to the bus. Thus, packets enter each SEC by having 
been written therein from other SFCs and not by reading them in. since read 
operations utilize the bus for signifirant amounts of time compared to write 
operations. Having the bus available generally whenever a SEC needs It helps to 

1 5 provide the full-wire throughput, 

However, when many SFCs write to- the same bus, the throughput is 
limited by the speed of the bus. 
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SUMMARY OF THE PRESENT INVENTION 

It is an object of the present invt^ntion to provide a cross-bar for 
communicating between Switching Ethernet Controllers (SEC) and PCIs. 

There Is Ifierefore provided in accordance with a preferred embodiment 
5 of the present invention. a data network including at least one crossbar, wherein 
each crossbar comprises N ports and a plurality N of devices each associated 
with and connected to one port of one of said crossbars. Each one port of one 
crossbar Includes an input buffer, a plurality N-1 of port output buffers, a 
plurality N-1 of fullness sensors, shutoff means 
10 The input buffer receives messages from the device connected to its port 

and sends said messages to the other ports of said one crossbar. Each port 
output buffers corresponds to one of said other ports, wherein each port output 
buffer receives said messages only from said input buffer of its associated othei 
port. Each fullness sensor is associated with one port output buffer and 
15 measures the fullness state of its associated port output buffer 

The shutoff means connects to the fullness sensors associated with the 
port output buffers corresponding to said one port at said N-1 other ports, for, 
when said fullness state for one of said other ports Is generally full, indicating to 
said device connected to said one port not to send data for the port which is 
20 now generally full. 

There is tiieiefore provided a network wherein each device additionally 
Includes N-1 device output buffers, one per the N-1 other ports of said crossbar. 

Additionally, there is therefore provided a network wherein each port 
comprises a bus link connected to said corresponding associated device, 
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Moreove^r, each device also includes a multiplicity of direct memory 
access (DMA) units for removing data from at least one of said device yulput 
butters. 

Furthermore, each crossbar In the network includes an arbiter for 
S providing said messages from said N-1*port output buffers to said device 
connected to its port only if said device is not full. 

There is therefore provided in accordance with a preferred embodiment 
of the present invention a switch fur a data networl^. the data network -including 
at least one crossbar having N ports and the switch being connectable to one of 
ig said N ports. The switch includes a multiplicity of switch output buffers, one per 
the N 1 other ports of said crossbar and ar least two direct memory access 
(DMA) units, each associated with at least one of said switch output buffers, for 
removing data frurn said associated at least one switch ou^ut buffers. 

15 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention will be understood and appreciated more fljily 
from the following detailed description taken in c»rijunction with the drawings in 
whicfK 

rig. 1 is a block diagram Illustration of a network of swrtching 
communiratfon controllers, constructed and operative in accordance with a 
preferred embodiment ot the present invention; 

Fig. 2 Is a schematic illustration of a switching communication cross bar 
controller fomning part of the network of Fig. 1; 
10 Fig, 3 10 a schematic illustration of a portion of the cross bar of Fig. 2 

and an Ethernet switching unitfonming part ot the switching network of Fig. 1: 

Fig. 4 Is a schematic Illustration of the switching unit and link connection 
forming pail of the switching network of Fig. 1; 

Fig. 5A is a flow chart illustration of the networking communication in 
IS accordance with a preferred embodiment of the present invenfion; 

Fig. 5B Is a illustration of a local link communication in accordance with 
switching Ethernet controller network of Fig. 1 ; 

Fig. 6A is a schematic diagram of an interface bus; 

Fig, 6B is a btod< illustration of a message architecture used in the 
20 interface bus of Fig . 6A; 

Fig. 6C Is a timing diagram illustraliun of the activity of the bus during the 
operations of Fig, 6A; and 

Fig, 7 is a block diagram illustration of the logical elements of a link 
message used In the present invention; 
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DETAILED DESCRIPTION OF THE PRESENT INVENTION 

Reference h now made to Figs- 1, 2, 3 and 4 which illustrate, in general 
terms, a cfoss bar 10 of the present invention and Its connection within a network, 
wherein Fig. 1 illustrates a general overview of one or more cross bars 10 

5 connected via one or more Individual buses 14 to one uc more Ethemet switches 
12. Since each Ethernet switch 12 is connected to the cross-bar 10 via its own 
individual btis 14, the cross bar 10 typically provides linking operations, 
transfening data from switch to svwtch. The one or more Ethernet switches^l?, are 
typically connected to one or more devices or work stations, not shown in the 

10 figures. 

As further illustrated in Fig. 1, the network switches 12 interconnect to 
create a large network or to enlarge an existing network, A plurality of network 
switches 12 are connected to PCI busses which are connected through 
PCI-to-PCI bridges. Thus, two bus networi^s can be connected together through 
i.s the addition of another PCI bus and two PCl-to-PCl bridges. 

Figs. 2, 3 and 4 illustrate the network of Fig 1, and specifically cross bar 
10 and elements of the associated switches 12, In some detail. In order to 
facilitate understanding, switches 12, along with other similar type elements, have 
been alphabetized to indicate location or sequence in the network. This 
20 numbering is for explanation only. In addition, in order to facilitate understanding, 
an-ows have been added to represent data packet flow; however, for clarity, not 
every data flow path has been mapped. 

As shown in Fig. 2. cross bar 10 comprises a multiplicity of ports 18 
where each port 16 comprises a link logic unit 18» an input FIFO buffer 20, a 
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plurality of output buffers 22 and a port transmit arbiter 26. Typically cross bar 10 
has four ports 1 6; however, it may comprise any number of ports 1 8, 

Each port 16, and the elements wfiidi il comprises, are dedicated to an 
assodated switch 12 or cioss bar 10, and are responaible for all communication 

5 with ite associated switch 12 or crossbar 10. Ae an example, the elements which 
comprise port 16A are associated with switch 12A and are responsible for switch 
12A's communication: link logic unit ISA receives and directs messages and data 
from switch 12A. and perfuims various port functions which will be described in 
more detail hereinbelow; input FIFO buffer 20A receives and stores data packets 

10 sent from switch 12A; output buffers 22 at port 16A receive and store data 
packets sent to switch 12A; and. port transmit arbiter 26A sends messages and 
data to switch 12A. 

Port to port communication is made via point-to point connection 
between input buffers 20 and their associated output buffers 22. As an example, 

15 and as represented in Fig, 2 by solid lines '^h, input buffer 20A Is connected to 
output buffers 22A. Output buffers 22A are located at ports 16B. 16C and 16D, 
respedively, 

Input buffer 20A transfers data packets via output buffers 22A to ports 
16B, 16C and 16D. Additionally, though not represented in the Figures, Input 
20 buffer 20B is connected to output buffers 22B. located at ports 16A, 160 and 16D, 
and so on. This poinWo-point connection allows simultaneous non-collision data 
transfers from the input buffers 20 to their dedicated output buffers 22, and hence 
simultaneous communication between ports 16. 

Communication starts ?3t switch 12, which writes a link message and/or 
25 an accompanying data packet inlo its associated link logic unit 18. Link messages 
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are either sent alone or interleaved with data packets. Two major typfisJ of link 
messages are possible: switch link messages to be transferred to another swittrfi, 
and local link messages indicating Ihe status of the associated switch 12. Switch 
link messages transferred with an accompanying data packet contain information 
3 conceming the associated data packet 

Logic unit 18 reads the first bit of the link message, whldi Identifies the 
type of link message being transmitted. As an exatiiple, if the first bit is set or 1, 
as an example, tlien logic unit 18 recognizes the message as a local link 
message, does not transfer the message, and proceeds to perform port functions 
10 which will bo described in more detail herelnbelow. If the first bit is not set or is 0, 
then the logic unit 18 recogni7e.s the link message as being a switcti link message 
and transfers the message and Uie associated packet to the input buffer 20. 

As an example, switch 12A sends a link message and data packet to 
logic unit ISA; logic unit 18A identifies the link message as a switch link message 
15 and transfers both the switch link message and the data packet to input buffer 
20A. 

Via liie direct point-to-point connections noted hereinabove, the Input 
buffer 20 on port 16 transfers the link message' and data packet to its associated 
output buffers 22 located on the other ports 16 on cross bar 10. As an example. 
20 input buffer 20A transfers the message and packet to output buffers 22A located 
on ports 16B, 16C and 16D. 

Each port 16 is identified by a port address, and each device, connected 
to one of the switches, is identified by its own device address or number. Each 
output buffer 22 comprises a device table register 24, coupled thereto. When port 
•)5 16 is linked to a switch 12. register 24 logically holds the device number of a 
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device linked to that specific port 16, When port 16 is linked to a cross bar 10, 
register 24 logically liolds the device numbers of ali the devices connected to the 
cro?w bar in llnkeci to that specific port 16 

The device nun^ber is also logically contained within the switch link 

^ message. When the switch link message and data packets are received at the 
port 16, the packet is first received by the device table register 24. If the device 
number data in the switch link message Is included in the receiving register 24, 
the message and packet is wfiUen into tiie coupled output buffer 22; if the device 
nuinber data in the switch link message Is not included in the receiving register 

10 24. then the register 24 simply Ignores the message, and does not receive It. 

As an example, the switch link message and its associated data packet 
of the previous example are Intended for transfer to a device linked to port 16B. 
Output buffeis 22A located on ports 16D, 16C and 16D receive the message and 
the packet The registers 24A at ports ieC and 16D do not include the device 

15 number contained in the switch message, and thus ignore the message. The 
register 24A coupled to output buffer 22A at port 16B does include the device 
number, and hence wril^^s Uie message and packet into output buffer 22A at port 
16B. Output buffer 22A at port 16B then transfers the message and packet to the 
port transmit arbiter 26B, which transfers the message and packet via link 148 to 

20 switch 12B. and onto its eventual destination. 

It is common that some switches 12 and their dedicated ports 16 are 
busier than others. For example, if switch 12A is excessively busy, the memory in 
switch 12A may become too full to receive more data packets. In such a case, 
the present invention imptements flow control which ensures thai no data is lost 

25 during the temporary backup. 
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In this situation, if switch 12A is full, it notifies sucli to logic unit 18A. 
Logic unit 18A then signals to port transmit arbiter 26A to cut off the data flow via 
link 14A to switch 12A. As a result, swilcfi 12A does not receive any more data 
packets, however, it does continue to send link messages and/or data packets to 
5 input buffer 20A. Once the memory in switch 12A is cleared, switch 12A sends a 
message to logic unit 18A to re-open up the link 14A and rf^^^llow data packet 
transfer to the switch. 

White Ihe link 14A to switch 12A is closed, all data packets sent to 
switch 12A are temporarily stored in the output buffers 22B, 22C. and 22D, 
10 located at port 16A. If port 16A continues to be busy, the temporarily stored data 
messages may backup in one of the output buffers 22 at port 16A, as an 
example, output buffer 22D at port IQA (Fig, 3). ' 

Each output buffer 22 comprises an almost full threshold 23, which 
when crossed, signifies that the output buffer 22 has become almost tuil and can 
15 not receive anymore data packets. Similarly, each output buffer 22 comprises an 
almost empty threshold 21. which when crossed, signifies Uial output buffer 22 
has become alrdost empty and can again receive data packets. 

In accordance with a preferred embodiment of the present invention, 
and as per the example from above, the data is output buffer 22D crosses almost 
20 full threshold 23. As indicated in Fig. 3 by dashed arrows 29. the ahnost full 
output buffer 22D (at port 16A) s^snds a message to the port transmit arbiter 26D 
(at port 16D). The message notifies port transmit arbiter 26D that switch 12 A is 
almost full and requests that port 16D stop sending data to switch 12A. 

Referring now to Fig. 4. each port transmit arbiter 26 comrnunicates with 
IS its associated switch 12 . via link 14 arid an associated switch arbite; 28, When 
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Iranemft arbitef 26 receives a signal to cut oft cutgorng data packets, it 
notifia<; fiuch to switch arbiter 28» which partially halts the oulward flow of data 
packtsls; when puil transmit artiiter 26 receives a signal to reopen communication, 
it notifies such to switch arbiter 28, which then reopens outward flow- 

5 As per the above example, port transmit arbiter 2BU notifies switch 

arbiter 28D of switch 121) {arrow 77) to stop transferring data to switdi 12A. 
Arbiter 28D stops transfer of data to switch 12A, however, still transfers data to 
otiier switches, as indicated by arrows 25. 

When the temporarily stored data packets at output buffer 2ZU (at port 

10 ISA) have cleared-out and have crossed the almost empty threshold 21, ttie 
output buffer ?7n notifies sends a message to bar arbiter 2QD (at port 16D) 
notijying it that switcli 12 A is almost empty. Port transmit arbiter 26D resumes 
sending data packets to port ISA. 

For the purpose of temporarily storing data packets while cross bar 

15 arisiter 26 has halted traffic, and as shown In Fig. 4. each switch 12 has a plurality 
of Direct Memory Access (DMA) units 30 and associated switch FIFO buffers 32. 
Eadi DMA 30 Is responsible for transfer of data from its associated FIFO buffer 
32. 

When port transmit arbiter 26n notifies arbiter 28D of switch 12D to stop 
20 sending data packets to switch 12A, switch arbiter 28D so indicates to the DMA 
unit 30A, DMA urul 30A stops transferring data from FIFO buffer 32A to switch 
12A. However, the remaining DMA units 30 remain active, sending data through 
port 160 to the other switches 12. 

While ttie outgoing data traffic from switch 12 is closed, data packets 
25 Intended for transfer are temporarily stored in switch buffers 32. As noted above. 
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em* C»» tfnted to an associated switch buffer 32. Additionally, each DMA 
30 is dedicated to one or more different ports 16. Generally each DMA 30 is 
dedicated to 2 ports, as example, ports 16A and 18D. however, DMA 30 could be 
dedicated to any number of ports depending on the switch's load, 

5 When arbiter 28 receives a message to close off outflow of data from 

switch 12 to an almost full port 16, only the outflow from the DMA 30 associated 
with the **full" port 10 is cut off. The DMA's 30 associated with the other ports 16 
remain active. Consequently, the only affected data flows are those of the 
associated ports and nqt the entire outgoing data flow from switch 12. 

10 Hence, in contrast to prior art network systems which required a total 

data flow halt upon collision from bacKups, the flow control system of the present 
Invention allows data flow to continue, and stems only that traffic affected by tlie 
backup. 

Reference is now made to Figs. 5A and 5B illustrating the 
H communication flow In the cross bar 10 among source switch 12A. port 16A on 
cross bar 10. port 16B on cross bar 10 and destination swilcli 12B. ^juinniariicing 
the flow discussed hereinabove. 

Switch 12A transfers (step 110) a link message and a data packet to 
logic unit ISA. The logic unit 18A identifies (step 112) the link message as a 
2u switch link m^ss^sage and writes (step 1 14) the link message and data packet into 
input buffer 20A, 

The Input buffer 20A transfers (step 116) the link message and data 
packet to the registers 24A at ports 16B, 16C and 16D. The registers 24A at 
ports 16B. 16C and 16D receive (step 118) the link message and the data packet 
25 transferred from switch 12A. Registers 24A at ports IOC and 1CD do not 
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recogni/e (step 120) the device number in the link message and ignore ffffi 
message and the data packet. The register 24A at port 16B, recognizes (sjep 
122) the device number in the link message, and the link message and data 
packet are written (step 124) Into the output buffer 22A at port 16B. 
s If switch 12B is not full, output bliffer 22A at port 16B transfers (step 

126) the data to port transmit arbiter 26B. which transfers (step 128) the message 
and data via link 14B to switch 12B. 

If switch 12B becomes full, it transfers (step 130, Fig. 68) a .local link 
message to logic unit 18B indicating such, Logic unit 18B indicates (step 132) to 
10 port transmit arbiter 2BB to stop sending any messages and/or data to switch 12B. 
Output buffer 22A at port 168 fills up (step 134) with temporarily stored 
unsendable messages and data. 

When output buffer 22A at port 16B reaches (step 136) almost fullness 
threshold 23. it notifies (step 138) such to port transmit arbiter 26A at port 16A. 
15 Arbiter 26A notifies (step 140) switch arbiter 28A at switch 12A not to send 
anymore data to port 10B until furtlier notice. Switch 12A indicates (step 142) 
to its DMA 30B (on switch 12A) to stop sending to switch 12B, and switch buffer 
32B (on switch 12A and dedicated to port 16B) temporarily stores (step 143) 
unsendable messages and data. 
20 Wlien switch 12B is capable of receiving again, il sends (step 144) a 

local link message to logic unit 18B Indicating that it is now open to receive. 
Logic unit 18B indicates (step 146) to port transmit arisiter 26B to reopen inflow to 
switch 12B. Output buffer 22A at port 16B restarts transmission (step 148). 
When the output buffer 22A at port 16B reaches an empty tlireshold 21 (step 150) 
25 it notifies such to port transmit arbiter 26A at port 1 6A. 
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Port transmit arbiter 2fc)A notifies (step 152) 55wlrch arbiter 28A at switch 
12A to resume transfer of data to poit 16B. Switdi A indiualeij (step 154) to DMA 
30B to resume sending data. Switch buffer 32B (on switch 12A and dedicated to 
port 16B) resumes (step 156) transfer of messages and data to switch 12B. 

5 Reference Is now made to Figs 8A, 6B. 6C and /. Figs. 6A, 6B and 6C 

describe a preferred embodiment of an Interface bus used in the individual bus 14 
of tfie present invention; Fig. 6A is schematic diagram of the interface bus, Fig. 6B 
is a message architecture used in the interface bus, and Fig, 6C is a timing 
diagram illustration of the activity of the bus during operations. 

10 Indivjdu;il bus 14 is a 17 bit, point-to-polni bus and comprises a dock 

signal 210. a command bit signal 212 and 16 bits of data signal 214. The 1G bit 
data 214 transfers either a 16 bit link message or data packets. Fig, 7 is a block 
diagram illustration of the logical elements of data bit 214, 

As noted hereinabove, each link 14 provides the connection and 

15 communication between one cross bar 10 and one switch 12, and transfers link 
messages and data packets therebetween. As additionally noted hereinabove, 
link messages 214 are transferred either alone or accompanied by data packets, 
and switch link messages transferred with data packets are used to communicate 
information about the associated data packet. 

20 Clock 210 functions in a manner similar to network clocks known in the 

art. and as such will not be described in detail herein. Command bit 212 is a on© 
cycle command word and data bits 214, depending on the command message, 
comprises betw^^en 0 to 33 cycles of 16 bit data words. In accordance with a 
piefened embodiment of the present invention, command messages signal the 
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commencement or end of a data packet When not transmitting B command 
message, command bit 212 transmits an Idle signal. 

Refem'ng now to Fig. OC, signals are constantly being transferred over 
links 14, cither in the form of commands, idle signals, link messages, or data 

5 packets. The timing signal rises to high with the initialization of a command bit 
212. remains at high for the duration of an Idle message, sinks from the high 
signal at the wiruTifcjncernefil of a link message/data packet and remains at low 
throughout the duration of the link message/data packet Hence, the rising or 
sinking state of the command bit Indicates whether to anticipate commands or to 

10 anticipate message/data. 

Referring now to Fig. 7, an exemplary protoool fur data packet 214 is 
shown which comprises 16 bits; bit 15 is a link message bit 220, bits 14-11 
provide high address bits 222, bits 10- 6 indicate a device number 224, and bite 
5-0 Indicate a message type 226. 

IS As noted hereinabove, the link message bit 220 is either set or not set. 

and is used to signify either a local link message or a switch link message, 
respectively. The high address 222 is provided for PCI address mapping and 
used for communication between switches 12 and PCIs connected to the network. 
1 he device number 9?A identififift the number of the device designated to receive 

20 the link message and/or dala packet Tlie message lype 226. as described in US 
patent application 03/790,155. which is incofporated herein by reference, relays 
messaging protocol between switches 12. 
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CLAIMS 

1 . A data-'oetwork comprising: 

at least one crossbar wherein each crossbar comprises N ports; 
a plurality N of devices each associated with and connected to one port 
of one of said crossbars; 

wherein each ono port of one crossbar comprises: 

an input buffer for receMng messages from the device 
connected to its port and for sending said messages to the N-1 other 
ports of said one crossban 

a plurality N -1 of port output buffers, each corresponding to one 
of said N-1 other ports, wherein each port output buffer receives said 
messages only froni said Input buffer of Ite associated other port; 

a plurality N-1 of fuihiess sensors, each associated with one 
port output buffer, for measuring the fullness state of its associated port 
output buffer; 

shutoff means, connected to the fullness sensors associated 
with the port oulpul buffers corresponding to said one port at said N-1 
other ports, for, when said fullness state for one of said other ports is 
generally full, indicating to said device connected to said one port not to 
send data tor the port which is now generally full. 
2, A network according to claim 1 wherein each device additionally 
comprises N-1 device output buffers, one per the N-1 other ports of said 
crossbar. 
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3, A n^itwork according to claim '? and whereirl each device aten comprisfis 
a inuIUpHcity of direct rnernury (DMA) uiiib for removing data 
from at least one of said device output buffers. 

4, A network according to claim 1 and wherein each croesbar comprises 
s an arbiter for providing said messages from said N-1 port output buffers 

to said device connected to Its port only If said device Is not full. 

5, A network according to any of the preceding claims, and wherein each 
port comprises a bus link connected to said corresponding associated 
device. 

10 6. A sswitch for a data network, the data network inctuding at least one 
u[0i>$bai liavint) N ports and the switch being conneclable to one of said 
N ports, the switch comprising: 

a multiplicity of switch output buffers, one per the N-1 other ports 
of said crossbar; and 

15 at least two direct memory access (DMA) units, each associated 

with at least one of said switch output buffers, for removing data from said 
associated at least one switch output buffers. 
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ABSTRACT 

A data network including al least one crossbar, wh(?reiti each crossbar 
comprises N ports and a plurality N of devices eaoli associated with and 
connected to one port of one of said crossbars. Each one port of one crossbar 

5 includes an input buffer, a plurality N-1 of port output buffers, a plurality N-1 of 
fullness sensors, shutoff means . The Input buffer receives messages from the 
device connected to its port and sends said messages to the other ports of said 
one crossbar. Each port output buffers corresponds to one of said other ports, 
wherein each port output buffer receives said messages only from said input 

10 buffer of its associated other port. Each fullness sensor Is associated with one 
port output buffer and measures the fullness stale of its associated port output 
buffer. 
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